Information processing device and recording medium

ABSTRACT

There is provided an information processing device including a recognition unit that recognizes a face area from a captured image, a storage controller that causes a storage unit to store face recognition information indicating a face area recognized by the recognition unit, and a display controller that superimposes onto a display unit a display indicating, as a face area candidate in a current captured image captured by an image capture unit according to an external instruction, an area that is among areas corresponding to a first face area indicated by face recognition information from another captured image stored in the storage unit, but is not included in a second face area recognized by the recognition unit from the current captured image, distinguishably from a display indicating the second face area.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of Japanese Priority PatentApplication JP 2012-279279 filed Dec. 21, 2012, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an information processing device and arecording medium.

Generally, when photographing multiple people, such as in a group photo,it is desirable to capture an image in which as many people as possiblehave a happy expression. For this reason, there is being developedtechnology that reports that everyone is smiling as a shutter timing byconducting smile detection or blink detection, for example, and an imagecapture device that, after image capturing, corrects a face area withouta happy expression with another image.

For example, Japanese Unexamined Patent Application Publication No.2008-198062 below discloses technology that takes a face area which,although recognized as a face area from a captured image, is not smilingor is looking away, and substitutes in a face area from another imagethat is smiling and looking forward.

SUMMARY

However, the technology disclosed in Japanese Unexamined PatentApplication Publication No. 2008-198062 above only processes areasrecognized as faces, and does not perform any processing on areas notrecognized as faces, such as faces looking sideways.

Consequently, in the present disclosure, there is proposed a new andimproved information processing device and a recording medium capable ofreports a face area not recognized as a face with image capture alone,on the basis of face recognition information from another image.

According to an embodiment of the present disclosure, there is providedan information processing device including a recognition unit thatrecognizes a face area from a captured image, a storage controller thatcauses a storage unit to store face recognition information indicating aface area recognized by the recognition unit, and a display controllerthat superimposes onto a display unit a display indicating, as a facearea candidate in a current captured image captured by an image captureunit according to an external instruction, an area that is among areascorresponding to a first face area indicated by face recognitioninformation from another captured image stored in the storage unit, butis not included in a second face area recognized by the recognition unitfrom the current captured image, distinguishably from a displayindicating the second face area.

According to an embodiment of the present disclosure, there is provideda non-transitory computer-readable recording medium having a programrecorded thereon for causing a computer to execute recognizing a facearea from a captured image, causing a storage unit to store facerecognition information indicating a recognized face area, andsuperimposing onto a display unit a display that indicates, as a facearea candidate in a current captured image captured by an image captureunit according to an external instruction, an area that is among areascorresponding to a first face area indicated by face recognitioninformation from another captured image stored in the storage unit, butis not included in a second face area recognized from the currentcaptured image, distinguishably from a display indicating the secondface area.

According to one or more of embodiments of the present disclosure asdescribed above, it is possible to report a face area not recognized asa face with image capture alone, on the basis of face recognitioninformation from another image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram illustrating an overview of an imageprocessor according to an embodiment of the present disclosure;

FIG. 2 is a block diagram illustrating a configuration of a cameraaccording to the first embodiment;

FIG. 3 is a flowchart illustrating operations of an image processoraccording to the first embodiment;

FIG. 4 is an explanatory diagram for explaining an associating processby a display controller according to the first embodiment;

FIG. 5 is an explanatory diagram for explaining an associating processaccording to a second algorithm by a display controller according to thefirst embodiment;

FIG. 6 is a flowchart illustrating operations of an image processoraccording to modification 1;

FIG. 7 is a flowchart illustrating operations of an image processoraccording to modification 2;

FIG. 8 is a flowchart illustrating operations of an image processoraccording to the second embodiment;

FIG. 9A is a flowchart illustrating operations of an image processoraccording to modification 3;

FIG. 9B is a flowchart illustrating operations of an image processoraccording to modification 3;

FIG. 10 is an explanatory diagram illustrating an overview of an imageprocessor according to the third embodiment; and

FIG. 11 is an explanatory diagram for explaining a compositing processby an image processor according to the third embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Hereinafter, the description will proceed in the following order:

1. Overview of information processing device according to embodiment ofpresent disclosure

2. Embodiments

-   -   2-1. First embodiment        -   2-1-1. Configuration        -   2-1-2. Operational process        -   2-1-3. Modification 1        -   2-1-4. Modification 2    -   2-2. Second embodiment        -   2-2-1. Operational process        -   2-2-2. Modification 3    -   2-3. Third embodiment

3. Conclusion

1. OVERVIEW OF INFORMATION PROCESSING DEVICE ACCORDING TO EMBODIMENT OFPRESENT DISCLOSURE

The present disclosure may be carried out in various embodiments,examples of which are described in detail from sections (2-1. Firstembodiment) to (2-3. Third embodiment). Also, an information processingdevice according to the present embodiment (information processingdevice 1) is equipped with:

A. a recognition unit (11) that recognizes a face area from a capturedimage;B. a storage controller (13) that causes a storage unit (2) to storeface recognition information indicating a face area recognized by therecognition unit; andC. a display controller (14) that superimposes onto a display unit adisplay indicating, as a face area candidate in a current captured imagecaptured by an image capture unit (4) according to externalinstructions, an area that is among areas corresponding to a first facearea indicated by face recognition information from another capturedimage stored in the storage unit, but is not included in a second facearea recognized by the recognition unit from the current captured image,distinguishably from a display indicating the second face area.

An image processor according to an embodiment of the present disclosuretakes, from a currently captured image captured by an image capture unitaccording to external instructions, an area in which, although facerecognition fails, a face area is determined to exist, and causes adisplay unit to display the area as a face area candidatedistinguishably from an image in which face recognition succeeded. Notethat in the present specification, the sensing of light by an imagesensor is designated image capture. For example, the image process isincluded in a device such as a fixed camera, a handheld camera held by auser, or a wearable camera attached to a user, animal, vehicle, or thelike.

For this reason, the recognition unit first conducts face recognition onan image captured continuously or at designated timings (the othercaptured image), and outputs face recognition information. Facerecognition information refers to information that includes informationindicating the position, range, capture time, smile level, or the likeof a recognized face area.

The storage controller then causes the storage unit to cumulativelystore face recognition information output from the recognition unit. Inthis way, by having the storage controller cause face recognitioninformation with a smaller data size compared to a captured image to bestored, it is possible to reduce the used storage capacity of thestorage unit over the case of causing captured images to be stored.

Subsequently, the recognition unit conducts face recognition on acaptured image (the current captured image) that the image capture unitcaptures according to external instructions, such as the press of ashutter button, or image capture instructions given via communication.

The display controller then determines a face area candidate in thecurrent captured image by associating the current captured image and theaccumulated face recognition information. More specifically, the displaycontroller determines a face area candidate by associating a face areaindicated by face recognition information with an area in which facerecognition failed from the current captured image. In other words, fromamong areas of the current captured image that correspond to a face arearecognized in another captured image (a first face area), the displaycontroller determines an area not included in an area in which facerecognition succeeded from the current captured image (a second facearea) as a face area candidate. For example, the display controllerdetermines, as a face area candidate, a face area of a subject for whom,although face recognition had been successful because the subject waslooking forward up until immediately before acquiring the currentcaptured image, face recognition now fails because the subject lookedsideways during the acquisition of the current captured image.Hereinafter, such a process of determining a face area candidate by thedisplay controller will also be designated the associating process.

After that, the display controller reports to the user by causing adisplay unit to superimpose a display indicating the determined facearea candidate onto the current captured image, distinguishably from adisplay indicating an area in which face recognition succeeded.

In this way, an image processor is able to use face recognitioninformation from another image to determine, as a face area candidate, aface area in which face recognition failed with the current capturedimage alone because a subject is looking sideways or the like, andreport the determined face area candidate to the user. For this reason,the image processor is able to clearly inform the user of the existenceof a subject for whom face recognition failed, and prompt the user toremind the subject to pay attention and acquire a current captured imageagain.

At this point, an overview of such operations for superimposing adisplay indicating a face area candidate by the image processor will bespecifically described with reference to FIG. 1. FIG. 1 is anexplanatory diagram illustrating an overview of an image processoraccording to an embodiment of the present disclosure. As illustrated inFIG. 1, the image processor 1 conducts an associating process on acurrent captured image 31, and causes a display unit to display anassociated current captured image 32.

More specifically, first, the image processor 1 may recognize the facesof subjects 92, 93, and 95 who are looking forward from the currentcaptured image 31. For this reason, as indicated in the image 31, theimage processor 1 superimposes confirmed recognition indicators 7, whichare displays indicating areas in which face recognition succeeded fromthe current captured image 31, onto the respective face areas of thesubjects 92, 93, and 95.

On the other hand, the image processor 1 fails to recognize the faces ofsubjects 91 and 94 who are not looking forward in the current capturedimage 31. Accordingly, by conducting the associating process discussedabove using pre-accumulated face recognition information, the imageprocessor 1 is capable of determining the face areas of the subjects 91and 92 as face area candidates. After that, as indicated in the currentcaptured image 32, the image processor 1 superimposes tentativerecognition indicators 8, which are displays indicating face areacandidates, onto the respective face areas of the subjects 91 and 92.Note that, as illustrated in FIG. 1, the tentative recognitionindicators 8 are displayed distinguishably from the confirmedrecognition indicators 7.

The above thus describes an overview of an image processor according toan embodiment of the present disclosure.

2. EMBODIMENTS 2-1. First Embodiment

[2-1-1. Configuration]

First, a configuration of a camera that includes an image processor 1according to the first embodiment will be described with reference toFIG. 2. Note that FIG. 2 is a block diagram illustrating a configurationof a camera 100 according to the first embodiment. As illustrated inFIG. 2, the camera 100 includes an image processor 1, a storage unit 2,a display unit 3, an image capture unit 4, an operating unit 5, and acommunication unit 6. Also, the respective structural elements of thecamera 100 are respectively connected via a bus 9. Hereinafter, eachstructural element of the camera 100 will be described in detail.

(Communication Unit)

The communication unit 6 is a module for communicating via a wired orwireless network such as a local area network (LAN) or phone line, andincludes functions for accepting external instructions. Specifically,the communication unit 6 accepts instructions such as image captureinstructions and image compositing instructions transmitted from anotherinformation processing device. Otherwise, the communication unit 6 mayalso transmit an image processed by the image processor 1 to anotherinformation processing device.

(Operating Unit)

The operating unit 5 includes functions for accepting externalinstructions. Specifically, the operating unit 5 accepts inputoperations such as image capture instructions and compositinginstructions from a user. For example, the operating unit 5 is realizedby a shutter button, or by a touch panel integrally formed with thedisplay unit 3.

(Image Capture Unit)

The image capture unit 4 includes an image sensor, image capture opticsthat include an image capture lens, and a captured image signalprocessor. The image capture unit 4 outputs data of a captured image inthe form of a digital signal. The image capture unit 4 captures an imageaccording to image capture instructions accepted by the operating unit 5or the communication unit 6, or under control by an image capturecontroller 15. Note that the image sensor is realized by acharge-coupled device (CCD) imager or a complementarymetal-oxide-semiconductor (CMOS) imager, for example.

(Display Unit)

The display unit 3 displays a current captured image captured by theimage capture unit 4 according to external image capture instructions,on the basis of control by a display controller 14. Otherwise, thedisplay unit 3 displays image continuously captured by the image captureunit 4 in real-time, without accepting external image captureinstructions. An image captured by the image capture unit 4 anddisplayed in real-time in this way is hereinafter also designated athrough-the-lens image. In addition, the display unit 3 is realized witha liquid crystal display (LCD) or organic light-emitting diode (OLED)display, for example.

(Storage Unit)

The storage unit 2 stores images captured by the image capture unit 4and face recognition information output from a recognition unit 11.Also, the storage unit 2 is realized with a recording medium such as amemory card or other flash memory, or a Digital Versatile Disc (DVD),for example.

(Image Processor)

The image processor 1 includes functions for conducting a face areacandidate determining process on a current captured image, andsuperimposing a tentative recognition indicator 8 indicating adetermined face area candidate onto the current captured image. Also, asillustrated in FIG. 2, the image processor 1 functions as a recognitionunit 11, an editing unit 12, a storage controller 13, a displaycontroller 14, and an image capture controller 15.

Recognition Unit 11

The recognition unit 11 recognizes a face area of a subject depicted ina captured image. Specifically, the recognition unit 11 recognizes asmile area, which is a face area of a smiling face, and a frontal facearea, which is a face area of a face looking forward. Furthermore, therecognition unit 11 is also able to recognize a smile level indicatingthe degree of a smile in a face area, and a frontal level indicating thedegree of coincidence between the angle of a face and the optical axisof the image capture unit 4. The recognition unit 11 then outputs theserecognition results as face recognition information.

Storage Controller 13

The storage controller 13 causes the storage unit 2 to store a currentcaptured image, as well as face recognition information output from therecognition unit 11. Also, the storage controller 13 causes the storageunit 2 to store association data output from the display controller 14discussed later. Herein, the storage controller 13 stores a currentcaptured image and association data in correspondence with the time atwhich the current captured image was captured. Also, the storagecontroller 13 stores face recognition information in correspondence withthe capture time of the image from which the recognition unit 11recognized that face recognition information. In addition, the storagecontroller 13 may also cause the storage unit 2 to store an imagecaptured by the image capture unit 4 for use in a process by the editingunit 12 in correspondence with the time at which that image wascaptured. Note that hereinafter, the act of the storage controller 13causing the storage unit 2 to store something will also be referred toas the storage controller 13 recording something.

Display Controller 14

The display controller 14 determines a face area candidate in a currentcaptured image according to the associating process discussed earlier,on the basis of accumulated face recognition information. At this point,the display controller 14 outputs, as an associating process result,association data indicating a position and range of a face areacandidate. In addition, the display controller 14 causes the displayunit 3 to superimpose a tentative recognition indicator 8 indicating theface area candidate onto the current captured image, distinguishablyfrom a confirmed recognition indicator 7 that indicates an arearecognized by the recognition unit 11. Note that hereinafter, the act ofthe display controller 14 causing the display unit 3 to displaysomething will also be referred to as the display controller 14displaying something.

Editing Unit 12

The editing unit 12 edits an area of a face area candidate in a currentcaptured image. For example, the editing unit 12 edits so as tosubstitute an area of a face area candidate in a current captured imagewith an area corresponding to the face area candidate in anothercaptured image that was captured and recorded at a time before or afterthe current captured image was captured. Additionally, the editing unit12 may also edit an area recognized as a face by the recognition unit 11from a current captured image.

Image Capture Controller 15

The image capture controller 15 includes functions for controlling theimage capture unit 4 to acquire a captured image. For example, the imagecapture controller 15 controls the image capture unit 4 according toimage capture instructions accepted by the operating unit 5 or thecommunication unit 6, and acquires a current captured image. Inaddition, the image capture controller 15 may also control the imagecapture unit 4 to continuously capture and acquire a through-the-lensimage, even without image capture instructions. Furthermore, the imagecapture controller 15 may also cause the image capture unit 4 to capturea captured image for use in an editing process by the editing unit 12.

(Supplemental Remarks)

The image processor 1, storage unit 2, display unit 3, image captureunit 4, and operating unit 5 discussed above may also form separateinformation processing devices that respectively include a communicationunit 6. In this case, the image processor 1 may conduct the associatingprocess and the superimposing of a tentative recognition indicator 8 bycontrolling the storage unit 2, the display unit 3, the image captureunit 4, and the operating unit 5 via a wired or wireless network.

The above thus describes respective configurations of a camera 100 andan image processor according to an embodiment of the present disclosure.

[2-1-2. Operational Process]

Next, operations of the image processor 1 will be described withreference to FIGS. 3 and 4. FIG. 3 is a flowchart illustratingoperations of an image processor 1 according to the first embodiment. Asillustrated in FIG. 3, first, in step S104 the recognition unit 11recognizes a face from a through-the-lens image, and outputs facerecognition information. Subsequently, in step S108, the storagecontroller 13 records the face recognition information output from therecognition unit 11. Then, in step S112 the image capture controller 15determines whether or not the operating unit 5 or the communication unit6 has accepted image capture instructions.

In the case where image capture instructions have not been accepted(S112/No), the process returns again to step S104. Thus, since stepsS104 to S112 are repeated, the storage controller 13 cumulativelyrecords face recognition information. Note that since the storagecontroller 13 records face recognition information with a smaller datasize compared to a captured image to be stored, it is possible to reducethe used storage capacity of the storage unit 2 over the case of causingcaptured images to be stored.

Herein, assume that the image processor 1 cumulatively records facerecognition information by repeating steps S104 to S112 even afteraccepting image capture instructions and ending the processing in stepS116 and thereafter to be discussed later. Thus, the image processor 1is able to record face recognition information without gaps, even in thecase where there are image capture instructions. In addition, the imageprocessor 1 is also able to conduct the associating process at someother time with an algorithm that differs from the image capture date,on the basis of recorded face recognition information.

Meanwhile, in the case where image capture instructions have beenaccepted (S112/Yes), in step S116 the image capture controller 15 causesthe image capture unit 4 to capture an image, and the storage controller13 records the captured current captured image. Subsequently, in stepS120 the recognition unit 11 recognizes a face from the current capturedimage, and outputs face recognition information. Then, in step S124 thestorage controller 13 records the face recognition information outputfrom the recognition unit 11.

Subsequently, in step S128 the display controller 14 conducts anassociating process on the current captured image and the accumulatedface recognition information. Note that the face recognition informationused in this associating process is the face recognition informationaccumulated in steps S104 to S108 at times before the current capturedimage was captured. The specific processing details of the associatingprocess will be discussed later.

Next, in step S132 the display controller 14 superimposes a tentativerecognition indicator 8 onto a face area candidate of the currentcaptured image, on the basis of the associating process result, which isassociation data indicating a position and range of a face areacandidate. Also, the display controller 14 superimposes a confirmedrecognition indicator 7 onto the current captured image, on the basis ofthe face recognition information for the current captured image that wasrecorded in the above step S124. For this reason, the display controller14 is able to clearly inform the user of the existence of a subject forwhom face recognition failed, and prompt the user to remind the subjectto pay attention and capture the image again.

Lastly, in step S136 the storage controller 13 records the associationdata. For this reason, at a later date, for example, the image processor1 is able to superimpose a tentative recognition indicator 8 on thebasis of association data recorded at the same time as the currentcaptured image. Similarly, the image processor 1 is able to superimposea confirmed recognition indicator 7 on the basis of face recognitioninformation recorded at the same time as the current image capture time.

Hereinafter, the associating process in step S128 will be specificallydescribed with reference to FIG. 4.

FIG. 4 is an explanatory diagram for explaining an associating processby a display controller 14 according to the first embodiment. Asillustrated in FIG. 4, the display controller 14 may superimposeconfirmed recognition indicators 7 on the basis of face recognitioninformation for images 33, 34, 35, 36, and 37.

Various algorithms are applicable to the associating process by thedisplay controller 14. Accordingly, in the present specification,examples of two algorithms will be described. These algorithms conductan associating process that treats a subject recognized with facerecognition information acquired at a time close to the time at which acurrent captured image was captured as existing continuously in the sameplace. Otherwise, the display controller 14 may also record motionstates of face areas indicating the same subject, and by predicting themotion, estimate the position of the face area when that subject's facefails to be recognized, for example.

(Associating Process According to First Algorithm)

The display controller 14 conducts the associating process andsuperimposes a tentative recognition indicator 8 in the case where thedifference between the time at which a current captured image wascaptured and the time at which face recognition information was acquiredis less than or equal to a first threshold value. Since the displaycontroller 14 uses accumulated face recognition information, the timetaken by the associating process is small compared to a comparativeexample in which images are accumulated and face recognition isconducted at the time of the associating process.

Hereinafter, the flow of an associating process according to the firstalgorithm will be described.

First, the display controller 14 selects face recognition informationfor an image that was captured before the time at which the currentcaptured image was captured, and for which the difference in capturetimes is less than or equal to a first threshold value. The displaycontroller 14 then conducts an associating process on the currentcaptured image and the selected face recognition information, anddetermines a face area candidate. In other words, the display controller14 treats a person whose recognized face is indicated by facerecognition information from immediately before the capture time of thecurrent captured image as also existing at the same place at the capturetime, and determines an area corresponding to a face area of that personas a face area candidate. After that, the display controller 14superimposes a tentative recognition indicator 8 onto the face areacandidate.

Next, a specific applied example treating the image 34 as the currentcaptured image will be described. At this point, assume that image datais recorded for the current captured image 34, whereas only facerecognition information is recorded for the images 33, 35, 36, and 37.

First, the display controller 14 selects the face recognitioninformation of the image 33 as the face recognition information of animage that was captured before the image 34 was captured, and for whichthe difference in capture times is less than or equal to a firstthreshold value. Subsequently, from among the face areas of the subjects91, 92, 94, and 95 indicated by the selected face recognitioninformation, the display controller 14 determines, as a face areacandidate, an area corresponding to the face area of the subject 91 whois not included among the face areas of the subjects 92, 94, and 95whose faces are recognized from the image 34. After that, the displaycontroller 14 superimposes a tentative recognition indicator 8A onto anarea corresponding to the face area of the subject 91, as illustrated inthe image 34-1.

The above thus describes an associating process according to the firstalgorithm.

(Associating Process According to Second Algorithm)

The display controller 14 conducts the associating process andsuperimposes a tentative recognition indicator 8 in the case ofsatisfying the following two conditions.

The first condition is that the difference between the time at which acurrent captured image was captured and the time at which facerecognition information was acquired is less than or equal to a secondthreshold value. Herein, the second threshold value is taken to belarger than the first threshold value in the first algorithm.Additionally, the first condition may also be that the difference isgreater than the first threshold value but less than or equal to thesecond threshold value. For this reason, the display controller 14 isable to conduct the associating process using face recognitioninformation with a more distant capture time that was not selected withthe first algorithm. However, in the case where the capture time isgreatly distant, it is conceivable that a subject has moved and is notpresent at the same place. Accordingly, in this algorithm, the followingsecond condition is set.

The second condition is that the number of matches between an areacorresponding to a face area indicated by selected face recognitioninformation and an area in which face recognition succeeded from acurrent captured image is equal to or greater than a third thresholdvalue. In other words, if the number of matching subjects whose facesare recognized is equal to or greater than the third threshold value,the display controller 14 conducts the associating process even if anamount of time equal to or greater than the first threshold value haspassed.

For example, in the case of capturing a group photo or the like, thesubjects are still, and thus most faces will continue to be recognized.Consequently, it is conceivable that there will be a large number ofmatching subjects whose faces are recognized. Accordingly, if the numberof matching subjects whose faces are recognized is equal to or greaterthan the third threshold value, the display controller 14 conducts theassociating process as though there were no subject motion, even if anamount of time equal to or greater than the first threshold value haspassed. In this way, with the second condition, the display controller14 is able to determine a face area candidate while accounting for thepossibility of subject motion.

Hereinafter, the flow of an associating process according to the secondalgorithm will be described with reference to FIG. 5.

FIG. 5 is an explanatory diagram for explaining an associating processaccording to a second algorithm by a display controller 14 according tothe first embodiment. As illustrated in FIG. 5, first, in step S128-1the display controller 14 calculates the difference between the capturetime indicated by the face recognition information, or in other wordsthe capture time of an image in which face recognition information wasrecognized, and the time at which a current captured image was captured.

Next, in step S128-2 the display controller 14 determines whether or notthe calculated difference is less than or equal to the second thresholdvalue, as discussed above as the first condition. At this point, asdiscussed earlier, the display controller 14 may also determine whetheror not the calculated difference is greater than the first thresholdvalue and also less than or equal to the second threshold value.

Subsequently, in the case where the difference is less than or equal tothe second threshold value (S128-2/Yes), in step S128-3 the displaycontroller 14 calculates the number of matches between an areacorresponding to a face area indicated by the face recognitioninformation and an area in which face recognition succeeded from thecurrent captured image.

Next, in step S128-4 the display controller 14 determines whether or notthe calculated number of matches is equal to or greater than the thirdthreshold value, as discussed above as the second condition.

Subsequently, in the case where the number of matches is equal to orgreater than the third threshold value (S128-4/Yes), in step S128-5 thedisplay controller 14 determines, as a face area candidate, an area inwhich face recognition failed from the current captured image, fromamong the areas of the current captured image that correspond to faceareas indicated by the face recognition information.

In step S132 as described earlier, the display controller 14superimposes a tentative recognition indicator 8 onto the areadetermined to be a face area candidate in this way.

Note that in the case where the difference between the time at which thecurrent captured image was captured and the time indicated by the facerecognition information is greater than the second threshold value(S128-2/No), the display controller 14 does not determine a face areacandidate and does not superimpose a tentative recognition indicator 8.Also, in the case where the number of matches between an areacorresponding to a face area indicated by the face recognitioninformation and an area in which face recognition succeeded from thecurrent captured image is less than the third threshold value(S128-4/No), the display controller 14 does not determine a face areacandidate, and does not superimpose a tentative recognition indicator 8.

For the second algorithm described above, a specific applied exampletreating the image 35 as the current captured image will now bedescribed. At this point, image data is recorded for the currentcaptured image 35, whereas only face recognition information is recordedfor the images 33, 34, 36, and 37.

First, assume that the display controller 14 determines that thedifference between the capture time indicated by the face recognitioninformation of the image 33 and the capture time of the image 35 isgreater than the first threshold value but less than or equal to thesecond threshold value. Herein, assume that there are three matchesbetween the face areas of the subjects 91, 92, 94, and 95 indicated bythe face recognition information of the image 33 and the face areas ofthe subjects 91, 92, and 95 whose faces were recognized from the image35, and that this value is equal to or greater than the third thresholdvalue. In this case, from among the face areas of the subjects 91, 92,94, and 95 indicated by the selected face recognition information, thedisplay controller 14 determines, as a face area candidate, an areacorresponding to the face area of the subject 94 who is not includedamong the face areas of the subjects 91, 92, and 95 whose faces arerecognized from the image 35. Subsequently, the display controller 14superimposes a tentative recognition indicator 8B onto an areacorresponding to the face area of the subject 94, as illustrated in theimage 35-1.

The above thus describes an associating process according to the secondalgorithm.

(Supplemental Remarks)

In the above, the display controller 14 conducts the associating processusing face recognition information from an image that was captured insteps S104 to S108 at a time before the current captured image wascaptured. However, the present disclosure is not limited to such anexample. For example, the display controller 14 may also conduct theassociating process using face recognition information from an imagethat was captured at a time after the current captured image wascaptured.

More specifically, after a current captured image is captured (that is,after the above step S124), the above steps S104 to S108 are repeated,and in the subsequent step S128, the display controller 14 may conductthe associating process using face recognition information accumulatedat that point. The display controller 14 may conduct the associatingprocess using face recognition information from both before and afterthe capturing of a current captured image, or conduct the associatingprocess using only one thereof.

Hereinafter, an example of conducting the associating process accordingto the first algorithm using face recognition information from an imagecaptured at a time after a current captured image is captured will bespecifically described, treating the image 36 as the current capturedimage. At this point, assume that image data is recorded for the currentcaptured image 36, whereas only face recognition information is recordedfor the images 33, 34, 35, and 37.

First, the display controller 14 selects the face recognitioninformation of the image 37 as the face recognition information of animage that was captured after the image 36 was captured, and for whichthe difference in capture times is less than or equal to the firstthreshold value. Next, from among the face areas of the subjects 91, 92,94, and 95 indicated by the selected face recognition information of theimage 37, the display controller 14 determines, as a face areacandidate, an area corresponding to the face area of the subject 94 whois not included among the face areas of the subjects 91, 92, and 95whose faces are recognized from the image 36. Subsequently, the displaycontroller 14 superimposes a tentative recognition indicator 8 onto anarea corresponding to the face area of the subject 94 in the image 36.

The display controller 14, by conducting associating processes accordingto the algorithms described above as an example, superimposes aconfirmed recognition indicator 7 or a tentative recognition indicator 8onto the subjects 91, 92, 93, 94, and 95.

The above thus describes an operational process of an image processor 1according to the first embodiment.

[2-1-3. Modification 1]

Next, a modification of the first embodiment will be described. An imageprocessor 1 according to the present modification is configured todisplay a tentative recognition indicator 8 in the case where thedisplay unit 3 displays a through-the-lens image irrespectively of imagecapture instructions. Hereinafter, operations of the image processor 1will be described with reference to FIG. 6.

FIG. 6 is a flowchart illustrating operations of an image processor 1according to modification 1. As illustrated in FIG. 6, first, in stepS204 the recognition unit 11 recognizes a face from a through-the-lensimage, and outputs face recognition information. Subsequently, in stepS208 the storage controller 13 records the face recognition informationoutput from the recognition unit 11.

Subsequently, in step S212 the display controller 14 conducts anassociating process on a through-the-lens image and accumulated facerecognition information. The associating process at this point is asdescribed above with reference to FIG. 4. After that, in step S216 thedisplay controller 14 superimposes a tentative recognition indicator 8onto a face area candidate in the through-the-lens image, on the basisof association data obtained in the above step S212. Thus, before theuser issues image capture instructions, the display controller 14 isable to clearly inform the user of the existence of a subject for whomface recognition failed, and prompt the user to remind the subject topay attention.

After that, in step S220 the image capture controller 15 determineswhether or not the operating unit 5 or the communication unit 6 hasaccepted image capture instructions. In the case where image captureinstructions have not been accepted (S220/No), the process returns againto step S204. On the other hand, in the case where image captureinstructions have been accepted (S220/Yes), in step S224 the imagecapture controller 15 causes the image capture unit 4 to capture animage, and the storage controller 13 records the captured currentcaptured image. Subsequently, in step S228 the recognition unit 11recognizes a face from the current captured image, and outputs facerecognition information. Lastly, in step S232 the storage controller 13records the face recognition information output from the recognitionunit 11.

The above thus describes an operational process of an image processor 1according to modification 1.

[2-1-4. Modification 2]

Next, another modification of the first embodiment will be described. Animage processor 1 according to the present modification does not make anew capture in response to image capture instructions, but records athrough-the-lens image at the timing of the image capture instructionsas a current captured image, together with association data from analready-conducted associating process. Hereinafter, operations of theimage processor 1 will be described with reference to FIG. 7.

FIG. 7 is a flowchart illustrating operations of an image processor 1according to modification 2. As illustrated in FIG. 7, in steps S304 toS320 the image processor 1 operates similarly as in steps S204 to S220described above.

Subsequently, in step S324 the storage controller 13 records theimmediately previous through-the-lens image. Then, in step S328 thestorage controller 13 records association data, which is the result ofthe immediately previous associating process in step S312, incorrespondence with the immediately previous through-the-lens image.

In this way, in the case of image capture instructions, the imageprocessor 1 does not duplicate the image capture and associatingprocesses.

The above thus describes an operational process of an image processor 1according to modification 2.

2-2. Second Embodiment

An image processor 1 according to the present embodiment performscompositing by substituting in an area corresponding to another imagewith respect to an area in which face recognition succeeded, or an areaof a face area candidate. Since the configuration of the presentembodiment is as described in the first embodiment, detailed descriptionwill be reduced or omitted herein. Hereinafter, operations of the imageprocessor 1 will be described with reference to FIG. 8.

[2-2-1. Operational Process]

FIG. 8 is a flowchart illustrating operations of an image processor 1according to the second embodiment. As illustrated in FIG. 8, first, instep S404 the recognition unit 11 recognizes a face from athrough-the-lens image, and outputs face recognition information.Herein, as discussed earlier, the face recognition information includesa smile level that indicates the degree of a smile in a recognized facearea. Subsequently, in step S408 the storage controller 13 records theface recognition information output from the recognition unit 11.

Subsequently, in step S412 the display controller 14 conducts anassociating process on a through-the-lens image and accumulated facerecognition information. The associating process at this point is asdescribed above with reference to FIG. 4.

After that, in step S416 the storage controller 13 records an image ofan area in which a face was recognized in the through-the-lens image,and whose smile level is higher than a designated threshold value, as asmile area image together with the position and range thereof.

Subsequently, in step S420 the display controller 14 displays anunacquired smile indicator, which indicates a face area in which a smilearea image has not been recorded yet, as well as an unacquired smilecount, which indicates the number of relevant face areas. Morespecifically, the display controller 14 superimposes an unacquired smileindicator onto an area in which a face was recognized or a face areacandidate in the through-the-lens image, and in which a smile area imagehas not been recorded. Additionally, the display controller 14 displays,as an unacquired smile count, the number of areas in which a smile areaimage has not been recorded from among the areas in which a face wasrecognized or the face area candidates in the through-the-lens image.For this reason, the display controller 14 is able to inform the userhow many more people are left to smile in order to capture an image witheveryone smiling.

Then, in step S424 the storage controller 13 determines whether or not asmile area image is recorded for all face areas of recognized face areasand face area candidates, and the image capture controller 15 determineswhether or not the operating unit 5 or the communication unit 6 hasaccepted image capture instructions.

In the case where a smile area image has not been recorded for all faceareas, and area image capture instructions have not been accepted(S424/No), the process returns again to step S404. Thus, the storagecontroller 13 is able to cumulatively record smile area images.Furthermore, the image capture controller 15 is also able to cause theimage capture unit 4 to repeatedly capture images until smile areaimages corresponding to all areas of recognized face areas and face areacandidates are recorded. Note that in the case of recognizing a smilearea with a higher smile level for the same face area, the storagecontroller 13 updates the already recorded smile area image. The storagecontroller 13 may also delete a face image from the storage unit 2 inthe case where a designated amount of time has elapsed since recording,or in the case of sensing subject motion with the associating process instep S412. Thus, the storage capacity of the storage unit 2 may bereduced.

On the other hand, in the case where a smile area image is recorded forall face areas, or in the case where image capture instructions havebeen accepted (S424/Yes), in step S248 the editing unit 12 compositesthe smile area images with the through-the-lens image and outputs theresult as the current captured image. More specifically, for the areasin which a face was recognized and the areas of face area candidates inthe current captured image, the editing unit 12 composites smile areaimages corresponding to the relevant areas. For this reason, the imageprocessor 1 is able to automatically generate a composite image in whichall or most of the subjects whose faces were recognized in the currentcaptured image, as well as the subjects whose faces were by coincidencenot recognized at that time, are smiling.

After that, in step S432 the storage controller 13 records the compositeimage output by the editing unit 12.

In this way, in cases such as when compositing by substituting in imagesof another, better expression for face areas without a good expressionin a current captured image, the image processor 1 is able topotentially substitute not only successfully recognized face areas, butalso face area candidates.

The above thus describes an operational process of an image processor 1according to the second embodiment.

[2-2-2. Modification 3]

Next, a modification of the second embodiment will be described. Animage processor 1 according to the present modification displays smilearea images corresponding to an area in which a face was recognized oran area of a face area candidate as candidate images selectable forcompositing, and composites on the basis of a user operation.Hereinafter, operations of the image processor 1 will be described withreference to FIGS. 9A and 9B.

FIGS. 9A and 9B are flowcharts illustrating operations of an imageprocessor 1 according to modification 3. As illustrated in FIG. 9A, insteps S504 to S520 the image processor 1 operates similarly as in stepsS204 to S220 described above.

Next, in steps S524 to S536 the image processor 1 operates similarly asin steps S116 to S128 described above. Then, in step S540 the storagecontroller 13 records association data in correspondence with a currentcaptured image.

After that, as illustrated in FIG. 9B, in step S544 the recognition unit11 recognizes a face from a through-the-lens image, and outputs facerecognition information. Subsequently, in step S548, the storagecontroller 13 records the face recognition information output from therecognition unit 11.

Then, in step S552 the storage controller 13 records an image of an areain which a face was recognized in the through-the-lens image, and whosedegree of smile is higher than a designated threshold value, as a smilearea image together with the position and range thereof.

After that, in step S556 the storage controller 13 determines whether ornot steps S544 to S552 have been repeated a designated number of times.In the case where steps S544 to S552 have not been repeated thedesignated number of times (S556/No), the process returns again to stepS544.

On the other hand, in the case where steps S544 to S552 have beenrepeated the designated number of times (S556/Yes), in step S560 thedisplay controller 14 superimposes a non-smile indicator, whichindicates a non-smile area, onto the current captured image. Morespecifically, on the basis of the association data output in step S536,the display controller 14 superimposes a non-smile indicator on anon-smile area not included among the smile areas from among the areasin which a face was recognized and face area candidates in the currentcaptured image.

Subsequently, in step S564 the display controller 14 superimposes faceimages which are smile area images accumulated in steps S544 to S552,and which correspond to a non-smile face area in the current capturedimage, as candidate images. For this reason, the user is able to selecta face image to composite from among candidate images. Herein, sinceface area candidates are areas in which a face was not even recognized,the question of a smile or not is not recognized either, and the facearea candidates are included among the non-smile areas. Accordingly, inthe case where a user looks at a current captured image and considers aface area candidate to be a smile, it is also possible to maintain thenatural state of the original face area without compositing.

Then, in step S568, the editing unit 12 composites candidate imagesselected by user selections into the current captured image, and outputsa composite image. Lastly, in step S572 the storage controller 13records the composite image output by the editing unit 12.

The above thus describes an operational process of an image processor 1according to modification 3.

2-3. Third Embodiment

Next, a third embodiment will be described. An image processor 1according to the present embodiment is installed in a server on anetwork, and composites images captured by multiple image capturedevices. More specifically, the editing unit 12 composites an areacorresponding to a face area candidate in an image captured by an imagecapture device different from the image capture device that captured acurrent captured image onto the area of the face area candidate in thecurrent captured image. Hereinafter, an overview of operations by animage processor 1 according to the present embodiment will be describedwith reference to FIGS. 10 and 11.

FIG. 10 is an explanatory diagram illustrating an overview of an imageprocessor 1 according to the third embodiment. As illustrated in FIG.10, installed cameras 108A to 108D are installed at fixed pointsthroughout an outdoor area. Hereinafter, the installed cameras 108A to108D will be collectively designated the installed cameras 108 when notbeing described individually. The present embodiment envisionssituations in which many installed cameras 108 are placed not onlyoutdoors as illustrated in FIG. 10, but also indoors. In addition, aserver 200 that includes the image processor 1 uses images captured bythe installed cameras 108 or a camera used by being worn on a user toconduct a face recognition process, an associating process, and thesuperimposing of various indicators.

The user 98 illustrated in FIG. 10 is capturing subjects 96 and 97 witha camera 104. A server 200 is connected to the camera 104 and installedcameras 108 via wireless communication. The camera 104 and the installedcameras 108 have the configuration of a camera 100 described above withreference to FIG. 2, but with the image processor 1 removed.

First, the camera 104 and the installed cameras 108 transmit capturedimages to the server 200. The server 200 then conducts a facerecognition process, the recording of face recognition information, andan associating process on the image received from the camera 104, andcauses a display unit of the camera 104 to display an image superimposedwith a tentative recognition indicator 8. Note that the camera 104 andthe installed cameras 108 are taken to transmit position informationindicating a current position together with an image to the server 200.

As illustrated in FIG. 10, although the subject 96 is facing in thedirection of the camera 104, the subject 97 is facing in the directionof the installed camera 108A rather than in the direction of the camera104. Consequently, whereas the face of the subject 96 is recognized, theface of the subject 97 is not recognized. For this reason, a confirmedrecognition indicator 7 is superimposed onto the subject 96 while atentative recognition indicator 8 is superimposed onto the subject 97 ina through-the-lens image.

At this point, operations of an image processor 1 included in the server200 for the case where a current captured image is captured according toimage capture instructions from the user 98 will be described withreference to FIG. 11.

FIG. 11 is an explanatory diagram for explaining a compositing processby an image processor 1 according to the third embodiment. Asillustrated in FIG. 11, the server 200 includes an image processor 1 anda communication unit 6.

When a current captured image is captured according to image captureinstructions from the user 98, the camera 104 transmits a currentcaptured image 38 to the server 200. Note that in the current capturedimage 38, the face of the subject 96 is recognized, but the face of thesubject 97 is not recognized. Meanwhile, the server 200 receives animage 39 captured by the installed camera 108A, which is an installedcamera 108 installed near the position of the subject 97 and installedwhere the subject 97 is facing forward. Note that the face of thesubject 97 is recognized in the image 39. Accordingly, the editing unit12 composites a face image of the subject 97 whose face was recognizedfrom the image 39 into a face area of the subject 97 given as a facearea candidate of the current captured image 38, and generates acomposite image 38-1.

In this way, the image processor 1 is able to conduct a compositingprocess with respect to a face area candidate of a current capturedimage, on the basis of an image obtained from a different image capturedevice.

The above thus describes an operational process of an image processor 1according to the third embodiment.

3. CONCLUSION

As described above, an image processor 1 according to an embodiment ofthe present disclosure is able to present to the user a face area, inwhich face recognition failed from a current captured image capturedaccording to image capture instructions, as a face area candidate,distinguishably from a face area in which face recognition succeeded. Inaddition, the image processor 1 is also able to composite a face imagecorresponding to another image into a face area candidate. Furthermore,the image processor 1 is able to composite a face image corresponding toanother image into an area where a face was recognized. Also, since facerecognition information successively accumulated by the image processor1 is recorded in correspondence with image capture times, the imageprocessor 1 is also able to apply, at a later date, an associatingprocess according to an algorithm with even higher face area candidateestimation precision.

The foregoing thus describes preferred embodiments of the presentdisclosure in detail and with reference to the attached drawings.However, the technical scope of the present disclosure is not limited tosuch examples. It is clear to persons ordinarily skilled in thetechnical field of the present disclosure that various modifications oralterations may occur insofar as they are within the scope of thetechnical ideas stated in the claims, and it is to be understood thatsuch modifications or alterations obviously belong to the technicalscope of the present disclosure.

For example, it is possible to create a computer program for causinghardware such as a CPU, ROM, and RAM built into an informationprocessing device to exhibit functions similar to each structuralelement of the above image processor 1. Also, a recording medium havingsuch a computer program recorded thereon is also provided.

Additionally, the present technology may also be configured as below:

(1) An information processing device including:

a recognition unit that recognizes a face area from a captured image;

a storage controller that causes a storage unit to store facerecognition information indicating a face area recognized by therecognition unit; and

a display controller that superimposes onto a display unit a displayindicating, as a face area candidate in a current captured imagecaptured by an image capture unit according to an external instruction,an area that is among areas corresponding to a first face area indicatedby face recognition information from another captured image stored inthe storage unit, but is not included in a second face area recognizedby the recognition unit from the current captured image, distinguishablyfrom a display indicating the second face area.

(2) The information processing device according to (1), wherein theinformation processing device further includes:

an editing unit that edits the face area candidate area.

(3) The information processing device according to (2),

wherein the storage controller causes a storage unit to store thecaptured image, and

wherein the editing unit composites, into the face area candidate area,an area corresponding to the face area candidate in a captured imagethat was captured and stored in the storage unit at a time before thecurrent captured image is captured.

(4) The information processing device according to (2),

wherein the storage controller causes a storage unit to store thecaptured image, and

wherein the editing unit composites, into the face area candidate area,an area corresponding to the face area candidate in a captured imagethat was captured and stored in the storage unit at a time after thecurrent captured image was captured.

(5) The information processing device according to any one of (2) to(4),

wherein the display controller superimposes a candidate image onto thedisplay unit, and

wherein the editing unit composites the candidate image into the facearea candidate area in the current captured image according to anexternal instruction.

(6) The information processing device according to any one of (2) to(5),

wherein the editing unit edits an area corresponding to the first facearea in the current captured image.

(7) The information processing device according to any one of (1) to(6),

wherein the display controller superimposes a display indicating theface area candidate onto a current captured image that is captured bythe image capture unit prior to the external instruction and displayedon the display unit in real-time.

(8) The information processing device according to any one of (1) to(7),

wherein the face recognition information from the another captured imageis face recognition information from a captured image that was capturedat a time before the current captured image is captured.

(9) The information processing device according to any one of (1) to(7),

wherein the face recognition information from the another captured imageis face recognition information from a captured image that was capturedat a time after the current captured image was captured.

(10) The information processing device according to any one of (1) to(9),

wherein the display controller superimposes onto the display unit adisplay indicating the face area candidate in a case in which adifference between a time at which the current captured image wascaptured and a time at which the another captured image was captured isless than or equal to a first threshold value.

(11) The information processing device according to any one of (1) to(10),

wherein the display controller superimposes onto the display unit adisplay indicating the face area candidate in a case in which adifference between a time at which the current captured image wascaptured and a time at which the another captured image was captured isless than or equal to a second threshold value, and the number ofmatches between the first face area and the second face area is greaterthan or equal to a third threshold value.

(12) The information processing device according to any one of (2) to(6),

wherein the information processing device further includes:

-   -   an image capture controller that controls the image capture        unit,

wherein the recognition unit recognizes a smile area from the capturedimage,

wherein the storage controller causes the storage unit to store an imageof a smile area recognized by the recognition unit,

wherein the image capture controller causes the image capture unit torepeat image capturing until images of smile areas corresponding to allof the second face area and the face area candidate area are stored inthe storage unit, and

wherein the editing unit composites images of smile areas stored in thestorage unit into the second face area and the face area candidate.

(13) The information processing device according to any one of (1) to(12),

wherein the recognition unit recognizes a smile area from the capturedimage, and

wherein the display controller superimposes onto the display unit adisplay indicating, as a non-smile area, an area in the current capturedimage that is among the second face area and the face area candidatearea, but is not included in a smile area recognized by the recognitionunit from the current captured image.

(14) The information processing device according to any one of (2) to(6),

wherein the editing unit composites, into the face area candidate area,an area corresponding to the face area candidate in a captured imagethat was captured by an image capture unit different from an imagecapture unit that captured the current captured image.

(15) A non-transitory computer-readable recording medium having aprogram recorded thereon for causing a computer to execute:

recognizing a face area from a captured image;

causing a storage unit to store face recognition information indicatinga recognized face area; and

superimposing onto a display unit a display that indicates, as a facearea candidate in a current captured image captured by an image captureunit according to an external instruction, an area that is among areascorresponding to a first face area indicated by face recognitioninformation from another captured image stored in the storage unit, butis not included in a second face area recognized from the currentcaptured image, distinguishably from a display indicating the secondface area.

What is claimed is:
 1. An information processing device comprising: arecognition unit that recognizes a face area from a captured image; astorage controller that causes a storage unit to store face recognitioninformation indicating a face area recognized by the recognition unit;and a display controller that superimposes onto a display unit a displayindicating, as a face area candidate in a current captured imagecaptured by an image capture unit according to an external instruction,an area that is among areas corresponding to a first face area indicatedby face recognition information from another captured image stored inthe storage unit, but is not included in a second face area recognizedby the recognition unit from the current captured image, distinguishablyfrom a display indicating the second face area.
 2. The informationprocessing device according to claim 1, wherein the informationprocessing device further comprises: an editing unit that edits the facearea candidate area.
 3. The information processing device according toclaim 2, wherein the storage controller causes a storage unit to storethe captured image, and wherein the editing unit composites, into theface area candidate area, an area corresponding to the face areacandidate in a captured image that was captured and stored in thestorage unit at a time before the current captured image is captured. 4.The information processing device according to claim 2, wherein thestorage controller causes a storage unit to store the captured image,and wherein the editing unit composites, into the face area candidatearea, an area corresponding to the face area candidate in a capturedimage that was captured and stored in the storage unit at a time afterthe current captured image was captured.
 5. The information processingdevice according to claim 2, wherein the display controller superimposesa candidate image onto the display unit, and wherein the editing unitcomposites the candidate image into the face area candidate area in thecurrent captured image according to an external instruction.
 6. Theinformation processing device according to claim 2, wherein the editingunit edits an area corresponding to the first face area in the currentcaptured image.
 7. The information processing device according to claim1, wherein the display controller superimposes a display indicating theface area candidate onto a current captured image that is captured bythe image capture unit prior to the external instruction and displayedon the display unit in real-time.
 8. The information processing deviceaccording to claim 1, wherein the face recognition information from theanother captured image is face recognition information from a capturedimage that was captured at a time before the current captured image iscaptured.
 9. The information processing device according to claim 1,wherein the face recognition information from the another captured imageis face recognition information from a captured image that was capturedat a time after the current captured image was captured.
 10. Theinformation processing device according to claim 1, wherein the displaycontroller superimposes onto the display unit a display indicating theface area candidate in a case in which a difference between a time atwhich the current captured image was captured and a time at which theanother captured image was captured is less than or equal to a firstthreshold value.
 11. The information processing device according toclaim 1, wherein the display controller superimposes onto the displayunit a display indicating the face area candidate in a case in which adifference between a time at which the current captured image wascaptured and a time at which the another captured image was captured isless than or equal to a second threshold value, and the number ofmatches between the first face area and the second face area is greaterthan or equal to a third threshold value.
 12. The information processingdevice according to claim 2, wherein the information processing devicefurther comprises: an image capture controller that controls the imagecapture unit, wherein the recognition unit recognizes a smile area fromthe captured image, wherein the storage controller causes the storageunit to store an image of a smile area recognized by the recognitionunit, wherein the image capture controller causes the image capture unitto repeat image capturing until images of smile areas corresponding toall of the second face area and the face area candidate area are storedin the storage unit, and wherein the editing unit composites images ofsmile areas stored in the storage unit into the second face area and theface area candidate.
 13. The information processing device according toclaim 1, wherein the recognition unit recognizes a smile area from thecaptured image, and wherein the display controller superimposes onto thedisplay unit a display indicating, as a non-smile area, an area in thecurrent captured image that is among the second face area and the facearea candidate area, but is not included in a smile area recognized bythe recognition unit from the current captured image.
 14. Theinformation processing device according to claim 2, wherein the editingunit composites, into the face area candidate area, an areacorresponding to the face area candidate in a captured image that wascaptured by an image capture unit different from an image capture unitthat captured the current captured image.
 15. A non-transitorycomputer-readable recording medium having a program recorded thereon forcausing a computer to execute: recognizing a face area from a capturedimage; causing a storage unit to store face recognition informationindicating a recognized face area; and superimposing onto a display unita display that indicates, as a face area candidate in a current capturedimage captured by an image capture unit according to an externalinstruction, an area that is among areas corresponding to a first facearea indicated by face recognition information from another capturedimage stored in the storage unit, but is not included in a second facearea recognized from the current captured image, distinguishably from adisplay indicating the second face area.